Storage devices intended primarily to provide persistent memory for computer databases are commonplace. Such devices include rotating disk drive stores and non-volatile, battery power-backed semiconductor memories.
In order to provide higher performance storage devices than those of previously developed solutions, extremely large arrays of semiconductor memories, known as SSD (solid state disk stores) have been used as storage devices on storage area networks.
An important application for SSD (solid state disk stores) is implementation of large databases to store representations of messages such as electronic mail or email exchanged between personal computers. However such databases can become very large and beyond the economic capacity of solid state stores.
Usage of such databases is commonplace. This has led to the development of efficient, scalable supporting software and equipment sometimes referred to as “messaging engines”. Some messaging engines may support, theoretically at least, many thousands of users on a single server computer or, typically, a messaging engine may support a single local group of intimately connected server computers or similar configurations.
However, problems and limitations have tended to impose an upper limit as to the number of uses that can be effectively deployed on a single server (computer) or server group (of computers). Such problems and limitations include issues related to availability and recovery, and hence, usability. Some commonly used messaging systems are generically referred to as “exchanges” (for example, together with appropriate computer equipment, the Microsoft® Exchange Server™ family of software products may implement, in part, a messaging system (using one or more exchange type databases).
In practice, the usefulness of exchange messaging systems can be greatly limited in that certain failures of an Exchange Server (i.e. a server computer supporting an exchange type database) may cause all of the many users of that particular Exchange Server to lose ready access to their messages. In such a situation, they may be unable to conveniently send or receive any email until the relevant Exchange Server has fully recovered from the failure.
Since the underlying exchange storage mechanism uses at least one complex and large database, full recovery has typically required a complete and non-corrupted restoration of an entire database. Such recovery may need to include reprocessing of any and/or all message transactions and messages that may have been received by the Exchange Server but which may not have previously been backed up by applicable means.
In such circumstances, to recover a lost or corrupted database, an administrator may initially reload an older copy of the database(s)—typically from the most recently generated complete backup available. Exchange software may then be directed to read logfiles (electronic journals and the like) that contain representations of email messages sent or received since the prior complete backup copy was created. Exchange related software might then update copies of those messages into the appropriate “mailboxes” in the database in order to update the database and thus bring it current.
However, since such recovery may involve the application of sequential logfile data records to pseudo-random (mailbox determined) “locations” in the database, recovery time may become great. Moreover, speed of recovery may be limited by the random-access performance of the deployed database storage devices (such as the commonplace SSD memories). Many storage devices for databases are not at all optimized for random access methods. Recovery time also depends on the sheer size of the database logfiles, and therefore may tend to be proportionate to the elapsed time and to the intensity of user activity each since the most recent full backup was completed.
The problems alluded to above may be exacerbated by the use of very large databases such as are possible to create on mechanical disk storage devices. Exchange databases can grow to be quire large, as messages accumulate in per-user data repositories such as those commonly known in the art as “Inboxes”.
In this context, it is useful to note that, for many users, email messages have a very limited period of active use. Emails received today may typically be interesting, and those of yesterday somewhat less so and so on. Consequently, most of the users' accesses may be to more recent email messages. Messages older than one week may be infrequently accessed, and their everyday value to a typical user may be largely archival in nature. Despite a rapid aging process, typical implementations of Exchange Servers store all email messages in a large database, or database group. This may lead to very large, and continually growing, databases. In such databases, email messages of various ages may occupy parts of the same database on storage devices having equally high-performance, with storage device allocation taking little account of likely frequency of access.
Such large, previously developed, exchange databases may be associated with a number of problems, some of which are described below.
In regards to recovery time: Large databases may be laborious to recover after a failure occurs, and recovery may become protracted. Depending upon the failure mode and the backup strategy deployed; a moderately sized enterprise exchange database may require an extended period such as hours or even days to recover. Use of large databases may result in an increased probability of recovery process failure prior to completion, thus introducing cascaded recovery issues and extended outage periods. Such events may have an adverse impact upon business activity dependent upon availability of email service.
In regards to storage cost: exchange databases are typically operated with fast (high-performance) storage devices in order to provide sufficiently quick response times to the users. Such high-performance devices may tend to be associated with higher costs. Fast storage devices may be needed both to ensure timely on-line responses to user requests and for supporting acceptably short recovery times. Whenever databases are permitted to grow very large, the cost of the database storage tends to become substantial and performance issues may arise.
In regards to backup contention: larger databases typically require longer back-up times. An enterprise may seek to avoid scheduled downtime for daily back-up processing by using a concurrent backup function such as may typically be provided by suppliers of exchange software. However, concurrent backup processes may interfere with timely responses to user requests and may act to reduce application performance especially in the case that an Exchange database is stored on rotating mechanical disk drive(s). Such performance impact may prevent concurrent backups during peak business hours. As a result, database logfiles may grow large and this may result in extended recovery times and increased risks such as of failure during a recovery itself.
In regards to recovery risk: Large databases are typically backed-up using removable mag. tape (magnetic tape) as a recording medium. Use of mag. tape requires good administrative management and provision for secure storage to contain the risks of misplacement, loss or failure of a backup medium. Loss or failure of backup media may prevent or hinder recovery of a secured exchange database. Large databases, long recovery times and complex procedures each exacerbate risk of database damage such as due to human error during an actual recovery procedure.
In regards to problems with quotas: in an attempt to control the burgeoning storage used for email messages on Exchange Servers, some administrators of Exchange Servers may impose a maximum size limitation upon per-user storage space (an upper bound on the size of each user's mailbox). Whenever such a policy is imposed, reaching a mailbox size limit may cause consequences for a user that may be annoying or harmful. To reinstate satisfactory operation a user may be forced to delete messages, or to select messages for archiving by software to one or more separate files (archives).
Desktop client software programs for PCs (Personal Computers) may be used with Exchange Servers. One such desktop client program, Microsoft® Outlook™, offers a capability for moving copies of emails from an exchange database on an Exchange Server to a local RDS (rotating disk storage) on the client user's own PC (the PC that runs his or her client program). Thus, a common method for achieving user quota compliance is for the user to use a client program archiving function to initiate the moving of some of the older messages from an exchange database to locally connected RDS on a client PC. In order to provide for an appropriate measure of resilience through redundancy, users may typically direct such files of archived messages be replicated using NAS (network attached storage) disks connected via a LAN (local area network, such as those embodied using IEEE 802.2 standards and colloquially known as Ethernet). IEEE is the well known Institute of Electrical and Electronic Engineers, which publishes technical standards and recommendations.
Such ad hoc archiving of copies of email messages may create management problems, and elevated costs and risks for the organization providing exchange services. For example, quotas may tend to lower user satisfaction as to the service provided, and user productivity may be reduced as a consequence of adding a burden (archiving and management tasks) to the workload of those users. Moreover, quotas may sometimes be only marginally effective at reducing database size. For example, a database that stores email messages for 5000 users and provides in the region of 50 Mbytes per user (equivalent to perhaps 30–45 days of email) may still require approximately a quarter of a Terabyte (250 Gigabytes or 2.0 E+12 bits) of storage device capacity.
Moreover, insufficient managed e-mail archives may increase legal risks and associated discovery costs. In legal proceedings, discovery processes may routinely compel a costly search through all existing e-mail archives. Uncontrolled e-mail archiving thus leads to huge legal searches that encompass not only Exchange Server databases and backup tapes but also, potentially, all the PCs in an entire enterprise. In some cases these legal searches have cost more than a million dollars.
In response to such risks and associated costs, some enterprises are moving to email message life-cycle management techniques. Businesses may contain control of email messages throughout their life cycle, by applying appropriate archival policies and deleting stored messages after expiration of a policy-defined archival interval.
Full backup is typically the method used for backing up an exchange database when using backup to, and recovery from, traditional storage devices such as mag. tape and RDS. Use of this method may require making a concurrent backup set of all exchange logfiles—and all of the contemporaneous exchange message stores. The inefficiencies of such an approach are readily apparent—since message stores can be extremely large, backup times can be protracted and require large capacity storage media such as mag. tape. The protracted time can easily become inconvenient and the probability of recording device failure may be substantially proportionate to data size. Moreover the consequential delays due to a recording failure may also be roughly proportionate to data size.
Similarly, on the recovery side, huge full backups may lead to long recovery times and increased probability of, and consequences of, failure.
Furthermore, the user of large capacity mag. tapes may require administrator intervention which may become, in turn, inconvenience and error prone. Thus, either procedural errors or media defects may cause a recovery process to manifest failure, either while data is being reinstated or when the recovery process subsequently runs a consistency check against the recovered database.
After backup restoration has been completed, a second part of recovery may be the restoration of transactions that have occurred after the last full backup. Email messages and other exchange transactions may typically be secured by writing representations thereof to log files (sometimes termed “logfiles”). Log files may be stored, for example, on designated RDS media. A log file may simply be a sequential record of unprocessed transactions in the order they occurred. Recovery processes may need to read each of these sequential transactions, and apply (typically write) each one to the appropriate message storage area (mailbox within the Exchange database) of the appropriate user. Other approaches to recovery and transaction journalizing may be applied to log file design.
In a scenario wherein a storage device holding an exchange database fails after a whole day's transaction have been completed, but prior to full database back-up, the server may be required in effect, to re-process the entire day's transactions. In previously developed solutions there may be no email service while this operation is in progress and users may be deprived of normal email service for an extended period.
Log files such as are used in exchange services are typically written and read using a SAM (sequential access method) which is a relatively fast method of using RDS. Applying logfiles to bring newly restored databases typically involves using the databases in a Random Access Method. Where Random Access Method is applied to RDS a great deal of physical read/write head movement is usually involved with consequential temporal inefficiency and overall low performance resulting. Consequently, the database storage disks may be a limited factor preventing rapid recovery of an interrupted exchange service. Random write performance of magnetic disk RDS may be limited to as few as 100 transactions per second and recovery times may be protracted.